An automatic system for recognizing fly courtship patterns via an image processing method

Fruit fly courtship behaviors composed of a series of actions have always been an important model for behavioral research. While most related studies have focused only on total courtship behaviors, specific courtship elements have often been underestimated. Identifying these courtship element details is extremely labor intensive and would largely benefit from an automatic recognition system. To address this issue, in this study, we established a vision-based fly courtship behavior recognition system. The system based on the proposed image processing methods can precisely distinguish body parts such as the head, thorax, and abdomen and automatically recognize specific courtship elements, including orientation, singing, attempted copulation, copulation and tapping, which was not detectable in previous studies. This system, which has high identity tracking accuracy (99.99%) and high behavioral element recognition rates (> 97.35%), can ensure correct identification even when flies completely overlap. Using this newly developed system, we investigated the total courtship time, and proportion, and transition of courtship elements in flies across different ages and found that male flies adjusted their courtship strategy in response to their physical condition. We also identified differences in courtship patterns between males with and without successful copulation. Our study therefore demonstrated how image processing methods can be applied to automatically recognize complex animal behaviors. The newly developed system will largely help us investigate the details of fly courtship in future research. Supplementary Information The online version contains supplementary material available at 10.1186/s12993-024-00231-4.

based on user preferences, and the following guidelines are provided for optimal configuration.
(1) Light Field Environment Contrast The suggested light field setting in this system is backlit, without specific requirements for intricate details on the fly's body.Consequently, the contrast between light and shadow is adjusted to such a level that the human eye can effectively distinguish between the background and the fly's body and legs.
(2) Video Format for Loading Given the diverse range of video decoding formats, the system presented in this study processes images.Users can record videos using their cameras, manually split all frames, input the frames per second (FPS) value into the system, and subsequently load all frames into the system.
(3) FPS Settings during Experimental Recording For optimal results, this study recommends an FPS setting of 24 or higher.The FPS camera setting used in this study was also 24.The accuracy of identity tracking is directly influenced by FPS.In cases of excessively low FPS, the continuity of fly movement and behavior becomes challenging to determine reliably.Considering the maximum moving speed of a male fly (18.24 mm/s) and the length of the fly (approximately 3.5 mm), if the movement distance between each frame is less than 1/3 of its body length, an FPS setting of 15.6 or above is advised.Users can adjust the FPS based on their specific requirements following this principle.
(4) Fly Size Parameter Requirements (e.g., Lengths and Width) This study employed flies with a length of 3.5 mm or above and a width of 1 mm.If users prefer smaller flies, they can achieve this by selecting a larger magnification lens.
Clear imaging of the fly's legs is essential for effective analysis.
(5) The Angle Definition for "Singing" Behavior Users have the flexibility to adjust these values in the program settings based on their specific circumstances.The definition of "singing" behavior, specifically the requirement of a 30-degree wing spread, is derived from the definition provided by (von Philipsborn et al., 2011;O'Sullivan et al., 2018).

Graphical User Interface Setting Guidelines:
Supplementary Figure 1 depicts the GUI interface during program execution.For the wing area, the intensity threshold was set at 80, the similarity index was set at 1.6, and the eccentricity of the torso was multiplied by 0.9.These values are based on empirical data, and users can directly modify them according to their preferences through the GUI interface.
(1) Wing Extraction Coefficient After obtaining the complete representation of the fruit fly body, the RGB color space of the fly body was divided by 80 and subsequently multiplied.After rounding, this process generally allows straightforward separation of the wing and torso areas.As the RGB color space values for the wings are significantly greater than those for the torso, users can adjust these values based on their specific imaging conditions.
(2) Similarity Coefficient A similarity index of 1.6 was used to identify the fruit fly based on the corresponding values for the head, thorax, and abdomen of the torso.A value closer to 2 indicates a closer correspondence of the torso.When the suggested baseline for this value is 1.6, users can customize it according to their specific requirements.
(3) Torso Eccentricity Threshold The eccentricity of the torso multiplied by 0.9 was used to determine whether a male fruit fly bent its body.When a male fruit fly bends its body, the shape of the torso deviates from an elliptical form, resulting in a decreased eccentricity.This metric is used to ascertain whether a male fruit fly has arched its torso, and users can set this threshold according to their specific preferences.
Users can easily load video data from a designated folder, set the threshold for background subtraction, and define the threshold for noise removal (areas smaller than the specified threshold are considered noise and are removed).Camera-related parameters such as the FPS during recording, the ratio between the image and the actual scale (number of pixels per mm), the similarity index, and the wing extraction coefficient can be configured.Finally, users can define mating behavior and specify the output folder to initiate the analysis.